DOCUMENT RESUME 



ED 410 539 



CS 012 900 



AUTHOR 

TITLE 



INSTITUTION 

SPONS AGENCY 

REPORT NO 
PUB DATE 
NOTE 

AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Campbell, Jay R. ; Donahue, Patricia L. 

Students Selecting Stories: The Effects of Choice in Reading 
Assessment. Results from "The NAEP Reader" Special Study of 
the 1994 National Assessment of Educational Progress. 
National Assessment of Educational Progress, Princeton, NJ. ; 
Educational Testing Service, Princeton, NJ. 

National Center for Education Statistics (ED) , Washington, 
DC. 

NCES-97-491 

1997-06-00 

106p . ; The title page title says "Special Study"; the cover 
title says "special Survey." 

National Library of Education, Office of Educational 
Research and improvement, U.S. Department of Education, 555 
New Jersey Avenue, N.W. , Washington, DC 20208-5641; phone: 
1-800-424-1616; World Wide Web: http://www.ed.gov/NCES/naep 
Reports - Research (143) 

MF01/PC05 Plus Postage. 

Comparative Analysis; Grade 12; Grade 8; *Reading 
Achievement; * Reading Material Selection; Reading Research; 
Secondary Education; Sex Differences; *Story Reading; 
*Student Attitudes; ^Student Evaluation 
^National Assessment of Educational Progress 



ABSTRACT 



Conducted as part of the 1994 National Assessment of 
Educational Progress (NAEP) , the "NAEP Reader" study was designed to compare 
the performance of students who were allowed to select a story with the 
performance of those who were assigned a story. One nationally representative 
sample of students at each grade was allowed to choose a story (from 7) to 
read. Distinct representative samples at each grade were assigned stories. 

All participants answered the same 11 comprehension questions. Results 
indicated: (1) no significant difference was observed among 12th- graders 

(and only a slight difference among 8th-graders) between the average reading 
scores of students who were given a choice of story and students who were 
assigned a story; (2) at grades 8 and 12, students in the choice group were 
more likely than students in the non-choice group to rate the assessment as 
easier than other tests or assignments that they had in school; (3) despite 
slight variations, the patterns of students' story selection were similar 
across racial/ethnic groups at both grades 8 and 12, and across gender groups 
at grade 8; (4) among 12th-graders , males were drawn to a story about a 

soldier and females were drawn to a story about a relationship; and (5) 
although identical questions were used, many questions were more or less 
difficult to answer in conjunction with certain stories. (Contains 14 tables 
and 10 figures of data. Appendixes contain data; the comprehension questions; 
and details of the procedures and methods used.) (RS) 
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What Is The Nation ’s Report Card? 

THE NATION’S REPORT CARD, the National Assessment of Educational Progress (NAEP), is the only nationally 
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making objective information on student performance available to policymakers at the national, state, and local levels, NAEP is 
an integral part of our nation’s evaluation of the condition and progress of education. Only information related to academic 
achievement is collected under this program. NAEP guarantees the privacy of individual students and their families. 

NAEP is a congressionally mandated project of the National Center for Education Statistics, the U.S. Department of 
Education. The Commissioner of Education Statistics is responsible, by law, for carrying out the NAEP project through 
competitive awards to qualified organizations. NAEP reports directly to the Commissioner, who is also responsible for providing 
continuing reviews, including validation studies and solicitation of public comment, on NAEP’s conduct and usefulness. 

In 1988, Congress established the National Assessment Governing Board (NAGB) to formulate policy guidelines for 
NAEP. The Board is responsible for selecting the subject areas to be assessed from among those included in the National 
Education Goals; for setting appropriate student performance levels; for developing assessment objectives and test specifications 
through a national consensus approach; for designing the assessment methodology; for developing guidelines for reporting and 
disseminating NAEP results; for developing standards and procedures for interstate, regional, and national comparisons; for 
determining the appropriateness of test items and ensuring they are free from bias; and for taking actions to improve the form and 
use of the National Assessment. 
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Introduction 



Students select stories to read in classrooms, in libraries, in their homes, and wherever 
reading materials are available. They choose different types of stories based on their 
purpose for reading and their personal interests. One situation where students rarely have 
a choice of what to read, however, is in a reading assessment. Some educators view this 
as a problem in assessing reading comprehension since students may be more engaged 
when they have chosen a text than when they are reading assigned texts. 

While the effect of choice on student performance in large-scale assessments and 
the psychometric ramifications of offering choice have been studied, The NAEP Reader 
study represents a significant departure from past efforts. Whereas earlier studies have 
focused on the effects of offering students a choice among test questions, 1 The NAEP 
Reader study was designed to examine the feasibility and measurement impact of offering 
test takers a choice of reading material on an assessment of reading comprehension. 

Conducted as part of the 1994 National Assessment of Educational Progress 
(NAEP), The NAEP Reader study was designed to compare the performance of students 
who were allowed to select a story with the performance of those who were assigned a 
story. Booklets containing a selection of seven stories were produced, one for grade 8 and 
a different selection for grade 12. One nationally representative sample of students at each 
grade was allowed to choose a story to read. Distinct representative samples at each grade 
were assigned stories. As The NAEP Reader study was administered in conjunction with 
the NAEP reading assessment, students participating in the study worked within the same 
50-minute time frame as students taking the main assessment. All participants, in both the 
choice and non-choice samples, answered the same eleven comprehension questions that 
were generically worded so as to be applicable to each and all of the stories. Students in 
the choice sample were given an additional question asking them to briefly explain the 
reason for their choice of story. The major findings from this special study are provided 
below. 



1 Lukhele, R„ Thissen, D. & Wainer, H. (1994). On the relative value of multiple-choice, constructed response, and 
examinee-selected items on two achievement tests. Journal of Educational Measurement, 31(3), 234-250. 
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Major Findings 



♦ Choice vs. Non-Choice Performance 

Among twelfth graders, no significant difference was observed between the average 
reading scores of students who were given a choice of story and students who were 
assigned a story. At grade 8, however, students who selected a story demonstrated slightly 
lower performance than students who did not have a choice of story. The difference was 
one scale-score point on a 0-to-100 scale with a standard deviation of 10. 

♦ Choice vs. Non-Choice Perceptions 

Some differences were observed in students’ perceptions of the assessment depending on 
whether or not they were allowed to choose a story. At both grades 8 and 12, students in 
the choice group were more likely than students in the non-choice group to rate the 
assessment as easier than other tests or assignments that they had had in school. Also, 
twelfth graders who could choose a story had higher estimations of their performance on 
the assessment than did their counterparts who were assigned a story. On the other hand, 
no significant differences between choice and non-choice groups were observed in 
students’ reports of their motivation for performing well on the assessment. 

♦ Patterns of Story Selection 

Despite some slight variations, the patterns of students’ story selections were mostly 
similar across racial/ethnic groups at both grades 8 and 12, and across gender groups at 
grade 8. Among twelfth graders, however, males and females demonstrated strikingly 
different story preferences. Males were predominantly drawn to a story about a soldier 
and females were predominantly drawn to a story about a relationship. 

♦ Story Selection Criteria 

The most frequently reported basis for story selection in both grades was an affective or 
general evaluative criterion. Also, twelfth graders were more likely than eighth graders to 
select a story because it represented a particular genre. 

♦ Context Effects of Stories on Comprehension Questions 

Although identical questions were used to assess students’ comprehension of each of the 
seven stories at each grade, there was evidence that many questions were more or less 
difficult to answer in conjunction with certain stories. 
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